UNCLASSIFIED 


Defense  Technical  Information  Center 
Compilation  Part  Notice 

ADP023735 

TITLE:  HPC  Enhancement  of  Plume  Modeling  for  Use  by  Military 
Simulators 

DISTRIBUTION:  Approved  for  public  release,  distribution  unlimited 


This  paper  is  part  of  the  following  report: 

TITLE:  Proceedings  of  the  HPCMP  Users  Group  Conference  2007.  High 
Performance  Computing  Modernization  Program:  A Bridge  to  Future 
Defense  held  18-21  June  2007  in  Pittsburgh,  Pennsylvania 

To  order  the  complete  compilation  report,  use:  ADA488707 

The  component  part  is  provided  here  to  allow  users  access  to  individually  authored  sections 
of  proceedings,  annals,  symposia,  etc.  However,  the  component  should  be  considered  within 
the  context  of  the  overall  compilation  report  and  not  as  a stand-alone  technical  report. 

The  following  component  part  numbers  comprise  the  compilation  report: 

ADP023728  thru  ADP023803 


UNCLASSIFIED 


HPC  Enhancement  of  Plume  Modeling  for  Use  by  Military  Simulators 
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{obenscha,  admoses}@lcp.nrl.navy.mil 


1.  Introduction  and  Background 

CT-Analyst®  is  a high  fidelity,  fast,  accurate,  model 
for  the  transport  and  dispersion  of  contaminants.  These 
capabilities  arise  from  the  use  of  Dispersion  Nomografs  , 
which  are  pre-computed  from  detailed  high-resolution 
three-dimensional  (3D)  computational  fluid  dynamics 
(CFD)  calculations  from  FAST3D-CT.  These 
calculations  are  run  ahead  of  time  and  currently  take  a 
week  or  more  to  run  on  a moderate  sized  high 
performance  computing  (HPC)  machine.  CT-Analyst 
runs  on  a variety  of  platforms  that  include  Windows,  Mac 
OS  X and  various  flavors  of  UNIX  and  Linux. 

An  application  programming  interface  (API)  was 
developed  to  make  CT-Analyst’s  high  fidelity  physics 
based  plume  predictions  available  to  modeling  and 
simulation  (M&S)  tools.  This  plume  API  was  used  with 
the  M&S  application  One  Semi-Automated  Forces 
(OneSAF).  OneSAF  is  a next  generation  entity  level 
simulation1'1.  For  a 10  by  5 km  region  of  Baghdad, 
OneSAF  made  use  of  CT-Analyst  through  the  application 
programming  interface  (API)  to  determine  the  plume 
locations,  the  plume  concentration  at  points  within  the 
simulation,  and  to  determine  the  attenuation  of  visibility 
along  a line.  Figure  1 shows  the  overall  process  required 
to  generate  these  plumes  for  OneSAF.  Before  the 
simulation  is  run,  FAST3D-CT  generates  a Pre-Computed 
Nomograf  database.  During  the  simulation,  OneSAF 
calls  a CT-Analyst  process  that  in  turn  interprets  the 
Nomograf  database  to  generate  plumes. 
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FAST3D-CT,  a detailed  physics-based  numerical 
model121,  was  developed  at  the  Naval  Research  Laboratory 
(NRL)  to  accurately  predict  plume  evolution  and  the 
contamination  footprints  resulting  from  these  releases. 
FAST3D-CT  is  a general-purpose  fully  3D  computational 
fluid  dynamics  model  for  contaminant  transport  in 
complex  urban  geometries.  It  is  based  on  the  high- 
resolution,  time  accurate  Flux-Corrected  Transport 
algorithms  invented  at  NRL.  A wide  range  of 
contaminant  physics  and  physical  environment  models  are 
employed  in  FAST3D-CT.  FAST3D-CT  is  used  to 
perform  the  time-accurate,  high-resolution  3D  CFD  as  an 
offline,  high-fidelity  urban  contaminant  transport  scenario 
generator. 

The  salient  data  and  statistical  features  from  the 
FAST3D-CT  CFD  calculations  are  summarized  and 
distilled  into  memory  efficient,  time-independent 
Dispersion  Nomograf™  data  sets.  These  are  interpreted 
and  evaluated  by  CT-Analyst.  There  are  no  time- 
dependent  integrations  performed  explicitly  by  CT- 
Analyst.  Instead,  all  predictions  produced  by  CT-Analyst 
are  simply  the  result  of  applying  an  interpolation 
procedure  utilizing  the  appropriate  Nomograf  data  set 
based  upon  the  high-resolution  CFD  results.  This  same 
interpolation  procedure  can  be  used  in  both  upwind  and 
downwind  directions  with  equal  effectiveness.  These 
simple  geometric  operations  are  used  to  determine  the 
probable  source  zone  upwind  of  each  sensor.  CT-Analyst 
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provides  the  unique  capability  to  immediately  backtrack 
and  simultaneously  determine  the  location  of  multiple 
unknown  sources  simply  based  on  sensor  readings  and 
meteorological  parameters. 

2.  CT-Analyst  API 

Since  CT-Analyst  3.0,  CT-Analyst  has  developed  a 
mature,  stable  API  that  allows  its  functionality  to  be 
extended  to  a variety  of  software  platforms  and  protocols. 
This  was  accomplished  by  adding  a middle  layer 
(Figure  2)  to  the  software  that  can  be  easily  mapped  to 
other  interfaces.  The  CT-Analyst  API  and  graphical  user 
interface  are  written  in  C++  using  Trolltech’s  QT 
Framework.  This  API  then  calls  the  Dispersion 

Nomograf  Library  functions  that  query  the  Dispersion 
Nomograf  Database.  The  Dispersion  Nomograf  Libraries 
are  primarily  written  in  FORTRAN.  Using  the  CT- 
Analyst  API,  CT-Analyst  has  exported  its  functionality  to 
other  protocols  such  as  Google  Earth’s  KML  file  format 
and  Microsoft  Component  Object  Model  (COM)[3!.  The 
COM  implementation  has  been  used  in  successfully  in 
other  software  packages  including  the  Missile  Defense 
Agency’s  PEGEM  and  Defense  Group,  Inc.’s  CoBRA®. 


Figure  2.  CT-Analyst  design  and  component 
breakdown 


3.  Implementation 

There  were  two  main  components  to  this  project: 
generating  the  Dispersion  Nomografs  for  the  area  of 
interest  and  developing  an  API  for  communication 
between  CT-Analyst  and  OneSAF. 

To  generate  the  Dispersion  Nomografs  for  the  area  of 
interest,  in  this  case  Baghdad,  the  typical  process  is:  1) 
acquire  the  geometry  data  for  the  area  as  well  as  other 
data  such  as  satellite  imagery,  2)  process  the  data  into  a 
model  compatible  with  FAST3DCT,  3)  run  the  relevant 
cases  with  this  model,  and  4)  generate  the  Nomograf 
database. 


The  original  input  data  for  the  Baghdad  geometry 
was  a 3D  OpenFlight  model  of  a 10  by  5 kilometer  area 
centered  on  the  US  ‘Green  Zone’.  This  OpenFlight  model 
contained  complete  renderings  of  the  Baghdad  landscape, 
including  terrain,  buildings,  roads  and  water.  This 
OpenFlight  model  was  then  converted  into  an  AutoCAD 
DXF  file,  which  expressed  the  scene  as  a two- 
dimensional  (2D)  collection  of  building  outline  polygons, 
with  a height  attribute  value  attached  to  each.  The  DXF 
information  was  then  translated  into  an  ESRI  shapefile,  a 
format  similar  to  DXF  that  is  our  preferred  format  for 
preparing  geometry  information  for  use  in  the  Nomograf 
generation  process. 

Once  the  building  height  information  was  collected, 
it  was  exported  from  the  polygon  format  into  a raster 
format  using  MF  Works,  a geographic  information  system 
raster  program  that  allows  for  the  direct  manipulation  of 
cell  values.  Two  terrain  models  were  generated,  one  with 
an  average  ground  height  (flat  earth)  and  one  with  the 
actual  terrain  values  from  the  OpenFlight  model.  A water 
model  was  created  using  previously  generated 

information.  Each  of  these  datasets  was  then 
georeferenced  using  the  Universal  Transverse  Mercator 
coordinate  system  and  exported  to  a format  compatible 
with  FAST3D-CT. 

Two  Dispersion  Nomograf  databases  were  generated, 
one  for  each  of  the  terrain  models.  Both  of  the  databases 
used  building  and  water  models.  Eighteen  separate 
FAST3D-CT  simulations  were  run  for  each  database. 
The  area  computed  was  a 10  by  5 km  area  at  a resolution 
of  6 meters.  This  was  by  far  the  largest  area  attempted  at 
this  resolution.  At  this  resolution,  any  areas  above  25  sq 
kilometers  require  HPC  resources  to  run  within  a 
reasonable  amount  of  time.  For  the  Nomograf  database 
with  flat  earth,  a 256  processor  SGI  Altix  at  NRL  was 
used  to  generate  the  database.  For  the  Nomograf  database 
with  terrain,  the  256  processor  SGI  Altix  at  NRL  was 
used  as  well  as  a 32  processor,  64  core  SGI  Altix  obtained 
through  the  High  Performance  Computing  Modernization 
Program  (HPCMP)  Dedicated  High  Performance 
Computing  Investment  program. 

To  communicate  with  OneSAF,  it  was  decided  with  a 
group  at  SAIC  who  are  implementing  Physics-based 
Environment  for  Urban  Operations  enhancements  to 
OneSAF  to  develop  a Java  API  for  CT-Analyst.  This 
approach  was  taken  because  Java  was  a common 
language  known  by  both  development  teams.  As  can  be 
seen  from  Figure  3,  when  OneSAF  requests  plume 
information,  a separate  process  developed  by  SAIC  would 
call  the  CT-Analyst  Java  API.  The  CT-Analyst  Java  API 
would  then  call  CT-Analyst  interface  via  the  Java  Native 
Interface  (JNI)  framework.  The  communication  between 
OneSAF  and  this  process  was  via  sockets.  The  API 
implemented  for  OneSAF  included  generating  plume 
density  contours,  concentrations  at  various  points  in  the 
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domain,  and  the  attenuation  of  visibility  due  to  source 
releases  between  two  locations  in  the  domain. 


1 ) Input  Data  2)  Geometry  Capture 


3)  FAST-3D  CFD  Runs 

Figure  3.  The  Nomograf  generation  process 

OneSAF  displays  plumes  as  polygons,  however,  CT- 
Analyst  computes  plume  displays  using  a raster  format. 
A function  was  developed  to  handle  this  translation  from 
a CT-Analyst  2D  concentration  array  to  polygons 
representing  concentration  contours.  The  first  version 
utilized  a basic  horizontal  and  vertical  raster  scan  that 
would  simply  detect  contour  points  by  searching  along 
each  row  in  the  concentration  array  for  the  concentration 
threshold.  The  second  version  improved  on  this 

algorithm  by  searching  only  for  the  first  value  where  the 
contour  threshold  was  reached,  and  then  performing  a 
neighbor-to-neighbor  search  until  the  contour  line  was 
complete.  This  dramatically  improved  the  accuracy  of  the 
generated  contours,  and  allowed  for  the  depiction  of 
concentration  gradients  within  them.  The  second  version 
also  allowed  multiple  sources  to  be  processed  at  once 
allowing  for  overlapping  plumes. 

The  point  concentration  commands  were 

implemented  by  first  calculating  the  plume  concentration 
for  the  sources  in  the  domain.  The  point  concentration 
was  then  read  from  the  calculated  concentration  array.  In 
a similar  manner,  the  attenuation  calculation  was 
implemented  by  integrating  the  concentration  from  the 
calculated  concentration  array  along  the  line  passed  in  by 
two  points. 

4.  Ongoing  Work 

Work  to  process  thousands  of  source  releases  per 
second  for  OneSAF  is  currently  underway.  The  most 
computationally  expensive  operation,  calculating  the 
plume  concentration  can  now  be  calculated  in  parallel 
using  Message  Passing  Interface  (MPI).  This  operation 


only  requires  each  node  to  be  passed  the  location  of  the 
source  in  UTM  coordinates  and  additional  properties  such 
as  mass. 

For  the  concentration  and  visibility  attenuation  calls, 
this  parallel  implementation  should  be  very  efficient 
because  the  only  additional  information  that  needs  to  be 
passed  to  each  node  is  the  location  of  points  to  be 
sampled  for  the  point  concentration  function  and  two 
points  per  line  for  the  attenuation  calculation.  With  the 
concentration  calculated  on  each  node,  the  single 
processor  implementation  can  be  used  to  calculate  the 
point  concentration  and  visibility  attenuation.  These 
values  can  then  be  gathered  using  a MP1REDUCE 
command.  At  this  time  the  generation  of  contour  plots  is 
still  an  expensive  operation  due  to  a serial  implementation 
of  the  raster  to  contour  generation. 

There  were  some  complications  with  using  Sun’s 
Java  Runtime  Environment  with  MPI.  On  several  MPI 
implementations  such  as  Myrinet’s  MPICH-GM,  the 
function  malloc  is  overloaded  with  MPICH-GM’s  own 
implementation  of  malloc  to  reduce  the  number  of 
memory  copies  per  operation.  This  conflicts  with  the 
version  of  malloc  supplied  by  Sun’s  Java  Runtime 
Environment.  However,  this  was  worked  around  by 
disabling  the  overriding  of  malloc  by  the  MPI 
implementation  or  using  another  variant  of  MPI  like  the 
standard  MPICH  framework  that  does  not  override 
malloc. 

5.  Results 

This  version  of  the  API  has  been  tested  on  several 
different  HPC  platforms  including  The  Army  Research 
Laboratory’s  x86  cluster powell  and  an  SGI  Altix  at  NRL. 
A third  HPC  platform  will  be  tested  in  the  near  future.  As 
can  be  seen  in  the  Figure  4,  OneSAF  original  plumes 
approximate  a plume  as  a circle  (Figure  4a)  while  the 
OneSAF  using  CT-Analyst  display  plumes  that  are 
affected  by  the  meteorological  conditions,  terrain  and 
buildings  in  the  domain  (Figure  4b).  Depending  on  the 
types  of  contaminant  in  the  plume,  this  can  have  a great 
affect  on  how  entities  within  the  domain  will  react.  For 
example,  when  combined  with  the  visibility  attenuation 
command,  an  entity  within  OneSAF  might  choose  to  go 
around  a plume  if  the  visibility  was  too  poor. 

6.  Future  Work 

The  benefit  of  accurate  plumes  can  be  seen  quite 
clearly  with  the  CT-Analyst  plume  displays.  Additional 
functions  such  as  CT-Analyst’s  unique  backtrack 
functionality  could  also  be  added  to  the  API  to  simulate 
unknown  source  detection  by  entities.  As  the  speed  of 
HPC  hardware  increases,  this  API  could  potentially  be 
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used  with  3D  CFD  codes  such  FAST3D-CT  instead  of 
CT-Analyst. 

This  work  was  supported  by  the  DoD  FIPCMP.  This 
work  was  supported  in  part  by  a grant  of  computer  time 
from  the  DoD  HPCMP  at  ARL. 


Figure  4.  OneSAF  depiction  of  plumes  using  their 
standard  plume  model  (4a)  vs.  CT-Analyst  plumes  (4b) 
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